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Description 

SYSTEM AND METHOD FOR COLOR 
ENHANCING IMAGES 

Background of Invention 
[0001] FIELD OF THE INVENTION 

[0002] Embodiments of the invention described herein pertain to 
the field of image analysis. More particularly, these em- 
bodiments enable color correction and enhancement of 
images. 

[0003] DESCRIPTION OF THE RELATED ART 

[0004] Existing methods of color image enhancement have relied 
upon automatic augmentation of a video signal, or have 
involved human controlled colorization of images. Inven- 
tions working on the video signal generally alter the 
chrominance portion of the signal without human inter- 
vention. These methods produce inferior results since the 
mapping of brightness values of an image to a hue re- 
quires human knowledge of what the hue of each object 



should be. For instance, with such methods, a human is 
required to subjectively intervene and make an educated 
guess as to the original hue of the region which is not 
possible by automated alteration of a video signal. Sys- 
tems that are human controlled involve using single or 
multiple base colors per region of an input image to map 
one luminance value of the input region to one resulting 
output color, yielding a one-to-one mapping. Regardless 
of the number of colors utilized within a region, these in- 
ventions still produce a one-to-one mapping per pixel of 
luminance to color. The resulting images enhanced by 
these methods lack an element of realism since they 
somehow appear too homogenous with respect to the 
range of colors produced. 
[0005] | n terms of color nomenclature used herein, there are 
multiple theoretical models of color in existence. These 
include the Munsell color-order system otherwise known 
as ANSI standard Z138.2, the RGB system, the CMYK sys- 
tem, the CIE color models, the HLS system and many oth- 
ers. The RGB system is used in video monitors and is an 
additive system wherein red (R), green (G) and blue (B) mix 
together to form the color white. The CMYK system used 
in printing applications and is a subtractive system 



wherein cyan (C), magenta (M), and yellow (Y) mix to ex- 
clude colors and form a color close to black, so black (K) 
is also used in printing to make pure black. The CIE color 
models are influential in color measurement applications. 
The HLS or hue luminance saturation system is defined by 
a double conical volume, the bottom apex being black, 
the top apex being white, the height along the axis defin- 
ing the luminance, the angle around the vertical axis 
defining the hue, and the distance from the axis defining 
the saturation. The HLS system is sometimes referred to 
as the HSL system, and has many closely related systems 
that are all more intuitive than picking RGB colors. Re- 
gardless of the theoretical model used, the transformation 
of one color to another color is a non-trivial process that 
involves the human eye, human mind, the electronic sig- 
nal used in transmitting the image, the specific camera 
and film type used, the paper and ink that the image is 
printed on, and many other factors that are highly com- 
plex. Lightness, brightness and luminance have complex 
relationships that are well known. 
[0006] | n u.S. Patent 4,984,072, entitled "System and method for 
color image enhancement", a method of color enhancing 
regions in images having similar desired hues is described 



in which color lookup tables are used in order to convert 
gray-scale values into unique values of hue, luminance 
and saturation. This method yields a one-to-one mapping 
within a region for each gray-scale value as the color 
lookup table is "predetermined" by the mapping of a 
gray-scale value in a region to a hue, luminance and satu- 
ration value. There is no variation to this one-to-one 
mapping. There are no means in which to add or subtract 
multiple base colors or in which to randomize colors. 
There are no means described for randomizing the colors 
or opacities of colors injected in patterns into regions. 
[0007] | n u.S. Patent 5,534,915, entitled "Method of color en- 
hancing a monochrome image using multiple base colors 
for selected regions of the monochrome image", a method 
is described in which multiple base colors are applied to a 
region of an image based on the brightness of each pixel 
in the region. This is accomplished by using a color 
lookup table that is constructed from multiple base colors 
defined for the input gray-scale value. Within that region, 
each gray-scale or luminance input value is mapped to 
one output color although different ranges of gray-scale 
can be mapped to entirely different colors. Therefore, 
there is still only a one-to-one mapping between gray- 



scale value in a region to a hue, luminance and saturation 

value. There are no means in which to utilize multiple 

base colors in any randomized or functional methodology. 

There are no means described for randomizing the colors 

or opacities of colors injected in patterns into regions. 
Summary of Invention 

[0008] Embodiments of the invention color-correct color images 
or movies, and colorize black and white images or movies. 
This is accomplished by the transformation of an input 
pixel luminance to an output color in a one-to-many 
mapping valid for a region within the image. Regions are 
selected for their desired output color, and are generally 
recognizable objects which are a subset of the main im- 
age. There are many methods of defining regions and em- 
bodiments of the invention can use a region resulting 
from the application of any such method. 

[0009] Colorizing an image or movie in a realistic manner com- 
prises selecting base colors and injection colors for pat- 
tern based insertion into regions of each image. Embodi- 
ments of the invention allow base and injection colors to 
be selected via color functions that may return more than 
one result per input luminance value. Injection colors may 
be additive or subtractive in nature when combined with 



the base colors. In addition, the opacity of the injected 
colors may be selected via alpha functions that determine 
how opaque an injected color will appear over the base 
color. Alpha and opacity are used interchangeably here- 
within. The injected colors are inserted into locations de- 
termined by a pattern function. This allows the resulting 
image to have regions which map luminance or gray-scale 
of a pixel to multiple colors in a one-to-many manner. 
These methods supply a broad palette of colors to the re- 
sulting image. The color, alpha and pattern functions may 
utilize luminance, position, or any other parameter in pro- 
ducing a return value. 
[0010] Base colors differ from injection colors in that they are 

additive, otherwise one would simply choose the comple- 
ment of the base color, and utilize a pattern that covers 
the entire region. Embodiments of the invention may set 
the base opacity to one minus the injection color opacity, 
while other embodiments use an alpha or opacity of one 
for the base color before mixing. The use of base colors 
allows colors to be easily added to a region without the 
need to specify the addition and subtraction setting, the 
opacity and without selecting a full area pattern. Other 
embodiments of the invention can bypass the use of base 



colors and allow the user to specify arithmetic mode, al- 
pha and pattern for all colors, some of which may be set 
to use full area patterns as base colors would, and some 
of which may be set to use sparse patterns for injection 
into the base color of the region. 

[001 1] a color function is a function that yields at least one color 
comprising a given input luminance. Embodiments of the 
invention can use any algorithm or mathematical function 
to generate output colors. Colors can be returned from a 
point, line, curve, surface or volume of color space, using 
any color space model, or from any combination of these. 
The volume can be the entire color system space or a 
subset of the volume of the color space. The resulting 
output image created from a color function has a level of 
realism that exceeds prior art methodologies. 

[0012] The injected colors can be added or subtracted from the 
base colors. Additive color math simply mixes the color 
with another color while subtractive color match mixes the 
complement of the color with another color. 

[0013] An alpha function returns an alpha value, which is a mea- 
sure of opacity for mixing. Any algorithm or mathematical 
function can be used in order to generate the resulting al- 
pha value including but not limited to return of a constant 



value, random value or random value within or outside of 
a range. 

[0014] a pattern function is any algorithm or mathematical func- 
tion that can be used in order to generate an "inject" or 
"no-inject" assertion based on an input location within the 
region, or using any other parameter. This is used in order 
to spread the injected colors throughout the region and 
provides for color non-uniformities that exist in real world 
objects. 

[0015] since there is no one-to-one mapping between input lu- 
minance and output color the resulting image appears 

deeper in color and texture. 
Brief Description of Drawings 

[0016] Figure 1A is a view of a color function space with a point 
constant in hue and saturation at two luminance values 
along with a hue in HLS space that varies in saturation and 
is shown at two different luminance values with the lumi- 
nance minimum value at the bottom of the figure and the 
luminance maximum at the top of the figure. 

[0017] Figure IB is a view of the hue path projected onto the lu- 
minance equator as seen from the luminance apex. 

[0018] Figure 2A is a view of a color function space with varying 
hue with a fixed saturation level in HLS space and is 



shown at two different luminance values. 

[0019] Figure 2B is a view of the saturation path projected onto 
the luminance equator. 

[0020] Figure 3A is a view of a color function space with varying 
hue and varying saturation in HLS space and is shown at 
two different luminance values. 

[0021] Figure 3B is a view of the varying hue and varying satura- 
tion in HLS space projected onto the luminance equator. 

[0022] Figure 4 is a view of an injection color function space that 
varies in hue and saturation at two different luminance 
values processed with additive and subtractive function, 
alpha function, pattern function and a base color at the 
same two luminance values combined via a mixer func- 
tion. 

[0023] Figure 5 is a view of two injection color function spaces 
which happen to overlap in the luminance axis showing 
four different luminance values processed with additive 
and subtractive function, alpha function, pattern function 
and a base color function space at the same four lumi- 
nance values combined by a mixer function. 

[0024] Figure 6 is a view of the flow chart of user actions when 
working with an embodiment of the invention. 

[0025] Figure 7 is a view of the "Apply to Region" task of Figure 6 



which shows an embodiment of the task that is performed 

for an input luminance value for a given pixel location 

within a region of an image. 
Detailed Description 

[0026] Embodiments of the invention color-correct color images 
or movies, and colorize black and white images or movies. 
This is accomplished in at least one instance by the trans- 
formation of an input pixel luminance to an output color 
in a one-to-many mapping valid for a region within the 
image. Regions are selected for their desired output color, 
and are generally recognizable objects which are a subset 
of the main image. There are many methods of defining 
regions and embodiments of the invention can use a re- 
gion resulting from the application of any such method. 

[0027] FIG. 1A is a view of a color function space using HLS space 
with luminance axis defined between points 104 and 103. 
Point 104 has luminance equal to zero, while point 103 
has luminance equal to one. The hue varies around the 
axis and the ratio of the distance of a point off of the axis 
to the distance to the outside of the double cone at that 
luminance value is the saturation which therefore varies 
from 0 (zero) to 1 (one). Point 100 constant in hue, satu- 
ration at luminance value 0.5 and point 107 with same 



hue and saturation value at luminance value 0.8 in HLS 
space. Point 103 in HLS space has luminance value 1, 
while point 104 has luminance value 0, while point 106 
has luminance value 0.5. Line 102 is constant in hue and 
varies in saturation and is shown at luminance value 0.5. 
Line 101 has the same range of saturation for the same 
hue as line 102, but at luminance value 0.8. Fig. IB shows 
the luminance plane 105 at luminance value 0.5 with 
point 100 and line 102 as seen from point 103, here- 
inafter referred to as the top of HLS space. Point 107 
projects onto point 100 and line 101 projects onto line 
102. Points 100 and 107 differ in luminance only, and 
lines 101 and lines 102 differ in luminance only. 
[0028] Figure 2A is a view of color function space with a varying 
hue with a fixed saturation level in HLS space and is 
shown at two different luminance values. Curve 202 is 
constant in saturation and varies in hue and is shown at 
luminance value 0.5. Curve 201 has the same range of 
hue for the same saturation as curve 202, but with lumi- 
nance value 0.8. Figure 2B shows the luminance plane 105 
at luminance value 0.5 with curve 202 as seen from the 
top of the HLS space. Curve 201 projects onto curve 202 
and differ in luminance only. 



[0029] Figure 3A is a view of a color function space with a varying 
hue and varying saturation in HLS space and is shown at 
two different luminance values. Area 302 varies in satura- 
tion and hue at luminance value 0.5. Area 301 has the 
same range of hue and saturation, but with luminance 
value 0.5. Figure 3B shows the luminance plane 105 at lu- 
minance value 0.5 with area 302 as seen from the top of 
the HLS space. Area 301 projects onto area 302 and differ 
in luminance only. 

[0030] Embodiments of the invention can use color functions that 
implement any algorithm or mathematical function to 
generate output colors including points as in Fig. 1A, lines 
in Fig. 1A, curves in Fig. 2A, areas in Figure 3A, or any 
combination thereof. In addition, the various geometric 
shapes are not required to remain constant through the 
luminance range, meaning that completely separate hue 
and saturation shapes or ranges may exist in different lu- 
minance ranges. Although the area in Figure 3A and 3B is 
shown appears as a circle, any geometric shape that varies 
in hue and saturation may be utilized. The use of HLS 
space is not a requirement and any color space may be 
utilized in specifying colors for use with different embodi- 
ments of the invention. 



[0031] Figure 4 is a view of injection color function space 400 

defining area 402 that varies in hue and saturation at lu- 
minance value 0.8 processed with additive and subtractive 
function 403, alpha function 404, pattern function 405, 
and base area 406 in base color function space 401 at the 
same luminance value 0.8 combined via mixer function 
407 to produce an output value. Injection area 409 and 
base area 410 are processed in the same manner as area 
402 and 406 although at luminance value 0.5. 

[0032] Multiple color spaces may be utilized for injection and 
base colors with each color space extending for part, or 
for the whole range, of input luminance values. This 
means that luminance ranges for different color functions 
may or may not overlap. Therefore, in addition to setting 
the color function luminance equator mapping to input 
luminance value, the upper and lower range of luminance 
values of a color function mapping may also be config- 
ured. This has the effect of shifting and scaling the result- 
ing output luminance value for a given input luminance 
value when applying the color function. 

[0033] Figure 5 is a view of two injection color spaces 400 and 
500 showing the hue and saturation area 402 at lumi- 
nance 0.85, area 409 at luminance value 0.6, areas 501 



and 502 at luminance value 0.3 and area 505 at lumi- 
nance value 0.15. Note that in this figure, the top of injec- 
tion color space 400 has luminance value of 1, while the 
bottom of injection color space 500 has luminance value 
0. The color functions yielding colors from areas 402, 
409, 501, 502 and 505 have these resulting injection col- 
ors combined respectively with the areas in base color 
space 401, namely area 406 at luminance 0.85, area 410 
at luminance value 0.6, area 503 at luminance value 0.3 
and area 507 at luminance value 0.15. Multiple base color 
spaces can also be utilized, but only one is shown for 
simplicity of drawing. Although color space 400 and 500 
are shown to overlap, they are not required to overlap. 
Area 501 is a second color area defined within color space 
400, while area 502 is defined within color space 502. As 
area 402 and 409 represent the hue and saturation values 
that one color function may combine with the luminance 
value to generate colors, area 501 shows that more than 
one color function may be chosen for color space 400 and 
the range of luminance values assigned to it. Areas 501, 
502 and 505 are shown without ray tracings to the top 
and bottom apex of their color space to represent func- 
tions that exist at only one luminance value. Area 501 and 



502 at identical luminance value may or may not be both 
asserted at the same output pixel location based on as- 
sertions from their respective pattern functions. 

[0034] S i nce areas 409, 410, 402, 406, 501, 502, 503, 505 and 
507 can comprise any geometric size or shape, they may 
be points, lines, curves, or any other geometric shape or 
combination thereof, and may vary in form throughout the 
luminance range, and therefore may not project onto the 
same shape on the luminance equator as seen from the 
apex. An embodiment of the invention may utilize only 
points in color space type HLS, while another embodiment 
may utilize generalized volumes throughout any color 
space type including but not limited to HLS, RGB, the vari- 
ous CIE spaces, CYMK, or any variation of HLS including 
HSV, HSB, or any other color space type. Embodiments 
may use brightness, lightness or luminance as an input to 
the color functions and calculations of these quantities are 
well known in the art for different color space types. 

[0035] For a given input luminance value, each injected color 
generation function returns a hue and saturation value 
pair that is either unaltered or complimented by additive 
and subtractive function 403. The alpha function 404 ap- 
plied to the color determines how opaque the injected 



color is, and may be set by the operator of the system to 
any value between 0 and 1 or any function that generates 
values in this range including a random value or a value 
outside of a given range or any combination thereof. The 
resulting color with assigned alpha or opacity is applied to 
the region based upon the result of pattern function 405. 
Any pattern returning assertions that yield entirely empty 
and entirely full areas or anything in between may be used 
and selected by the operator. This includes random return 
of assertion based on the input location of the pixel un- 
dergoing processing, or repeating patterns such as true, 
true, false repeated over and over for each successively 
located pixel. Any function may be used including more 
complex functions that return polka dot shaped patterns, 
or any other pattern known including texture maps that 
repeat any number of times. Use of a texture map may 
comprise setting threshold values with luminance, posi- 
tion or any other parameter in order to determine whether 
to assert the injection or not based on the texture map 
values. 

[0036] Base colors at luminance values that correspond to injec- 
tion colors are combined by a mixer function 407. The 
application of additive and subtractive function 403, alpha 



function 404 and pattern function 405 occurs in no par- 
ticular order, as different embodiments of the invention 
may apply a pattern function first in order to determine 
whether to generate a color before rendering its comple- 
ment or opacity. 

[0037] Although two injection color spaces are shown for sim- 
plicity, multiple color spaces for both injection color and 
multiple color spaces for base color may be utilized. 

[0038] Although base color function space 401 shown in Fig. 4 is 
identical to base color function space 401 shown in Fig. 5, 
i.e. the same hue and saturation ranges are returned 
along the same paths such as area 406 and 410, each of 
the functions have been selected to map their values for 
different input ranges of luminance. In this case, the be- 
low average luminance colors map to more of the input 
luminance range so the base colors in this case are 
slightly darker than if the base color function space equa- 
tor was equidistant between upper and lower apex as is 
the case in Fig. 4. 

[0039] Associating a range of luminance values in which to map a 
color function space to can involve clipping a color func- 
tion space, stretching the color function space, overlaying 
another space or use of any other mathematical function 



which can map an input luminance value to an output lu- 
minance value. 

[0040] pig. 6 shows a flow chart of actions that a user undertakes 
when working with an embodiment of the invention. The 
user selects base color (e.g., step 600) and/or injection 
color (e.g., step 602) functions to apply to a given region. 
The functions can be chosen from a list, drawn on a color 
wheel, chosen from a mathematical functions list, or via 
any other method. The color wheel may show the allow- 
able hue and saturation values for the mapped value of 
input luminance for an embodiment of the invention uti- 
lizing the HLS color system. The base and injection color 
functions are independently associated with a range of in- 
put luminance values (e.g., steps 604 and 606 respec- 
tively). The injection pattern is selected (e.g., at step 608) 
from a list or by inputting a mathematical formula or by 
any other means. The alpha function is selected (e.g., at 
step 610) from a list or by inputting a mathematical for- 
mula or by any other means. At step 612, the arithmetic 
mode is selected for injection processing. Additional base 
and injection colors may be added at steps 614 and 616. 
The inputs selected are applied to the region (e.g., step 
618). Other embodiments of the invention may show the 



effects of altering the settings after the user has input 
each one or during the time when the user is inputting or 
altering the settings. One skilled in the art would recog- 
nize that the order of these tasks may be altered in any 
way which produces the same result. 
[0041] pig. 7 shows a flow chart detailing the "Apply to Region" 
task of Fig. 6. A base color function is calculated (e.g., at 
step 700) for a given input luminance value for a given 
pixel location within a region of an image. This allows 
multiple values of color to result from one input lumi- 
nance value. An injection pattern is calculated (e.g., at 
step 702) and if the function determines that an injection 
color should be asserted at a given location and lumi- 
nance value (e.g. step 704) then the injection color func- 
tion is calculated (e.g., step 706). The injection alpha 
function is calculated (e.g., at step 708) and the depend- 
ing on the arithmetic mode, complemented or left unal- 
tered (e.g., step 710). Any and all base and injection col- 
ors calculated are mixed (e.g., step 712). This process re- 
peats based on the luminance value of every pixel in the 
region (e.g., step 714). One skilled in the art would recog- 
nize that the order of these tasks may be altered in any 
way which produces the same result. Thus, the sequence 



of the flowcharts described herein should not be viewed 
as being limited to any particular order of events. 
[0042] Computer systems capable of processing the amount of 
data commonly found in a region or image are readily 
available and networked systems wherein each separate 
processor works on a portion of the image are also avail- 
able and are capable of hosting embodiments of the in- 
vention. 



